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Preface  &  Acknowledgements 


Welcome  to  our  Tenth  Annual  Acquisition  Research  Symposium!  We  regret  that  this 
year  it  will  be  a  “paper  only”  event.  The  double  whammy  of  sequestration  and  a  continuing 
resolution,  with  the  attendant  restrictions  on  travel  and  conferences,  created  too  much 
uncertainty  to  properly  stage  the  event.  We  will  miss  the  dialogue  with  our  acquisition 
colleagues  and  the  opportunity  for  all  our  researchers  to  present  their  work.  However,  we 
intend  to  simulate  the  symposium  as  best  we  can,  and  these  Proceedings  present  an 
opportunity  for  the  papers  to  be  published  just  as  if  they  had  been  delivered.  In  any  case,  we 
will  have  a  rich  store  of  papers  to  draw  from  for  next  year’s  event  scheduled  for  May  14-15, 
2014! 


Despite  these  temporary  setbacks,  our  Acquisition  Research  Program  (ARP)  here  at 
the  Naval  Postgraduate  School  (NPS)  continues  at  a  normal  pace.  Since  the  ARP’s 
founding  in  2003,  over  1 ,200  original  research  reports  have  been  added  to  the  acquisition 
body  of  knowledge.  We  continue  to  add  to  that  library,  located  online  at 
www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  70  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  encourage  your  future  participation. 

Unfortunately,  what  will  be  missing  this  year  is  the  active  participation  and 
networking  that  has  been  the  hallmark  of  previous  symposia.  By  purposely  limiting 
attendance  to  350  people,  we  encourage  just  that.  This  forum  remains  unique  in  its  effort  to 
bring  scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  It  provides  the  opportunity  to  interact  with  many  top  DoD 
acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both  in  the  formal 
panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals,  breaks,  and  the 
day-ending  socials.  Many  of  our  researchers  use  these  occasions  to  establish  new  teaming 
arrangements  for  future  research  work.  Despite  the  fact  that  we  will  not  be  gathered 
together  to  reap  the  above-listed  benefits,  the  ARP  will  endeavor  to  stimulate  this  dialogue 
through  various  means  throughout  the  year  as  we  interact  with  our  researchers  and  DoD 
officials. 

Affordability  remains  a  major  focus  in  the  DoD  acquisition  world  and  will  no  doubt  get 
even  more  attention  as  the  sequestration  outcomes  unfold.  It  is  a  central  tenet  of  the  DoD’s 
Better  Buying  Power  initiatives,  which  continue  to  evolve  as  the  DoD  finds  which  of  them 
work  and  which  do  not.  This  suggests  that  research  with  a  focus  on  affordability  will  be  of 
great  interest  to  the  DoD  leadership  in  the  year  to  come.  Whether  you’re  a  practitioner  or 
scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 
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Abstract 

DoD  spending  on  services  has  been  trending  upwards  for  over  a  decade  and,  as  of  201 1 ,  it 
accounted  for  56%  of  total  contract  spending.  The  increased  reliance  on  services  contractors 
has  prompted  the  GAO  to  look  more  closely  at  the  acquisition  and  contract  management 
process.  In  this  research,  we  address  the  following  questions:  (1)  How  do  different 
stakeholders  define  successful  services  contracts  within  the  Navy?  (2)  How  do  different 
stakeholders  measure  services  contracts  within  the  Navy?  and  (3)  How  should  Navy  services 
contracts  be  defined  and  measured?  We  conducted  a  survey  of  168  key  stakeholders.  We 
discovered  that  when  defining  and  measuring  the  success  of  a  service  contract,  all 
stakeholders  tend  to  utilize  outcome-related  factors  over  process-oriented  factors.  We  believe 
this  is  because  outcomes  tend  to  drive  perceptions  of  success  more  than  processes  and  are 
more  easily  quantifiable.  Metrics  used  to  measure  success  are  typically  related  to  cost, 
schedule,  and  performance.  Based  on  these  findings,  we  provide  recommendations  on 
establishing  better  internal  control  measures,  putting  in  place  an  operational  audit  process, 
and  creating  a  standardized  reporting  process. 

Introduction 

The  service  sector  represents  the  largest  and  the  fastest  growing  segment  of  the 
economies  of  the  U.S.  and  other  developed  countries.  This  growth  of  services  in  the  overall 
economy  is  also  mirrored  by  the  growth  of  services  acquisition  in  the  DoD.  For  example,  the 
DoD  obligations  on  contracts  have  more  than  doubled  between  fiscal  years  2001  and  2008 
to  over  $387  billion,  with  over  $200  billion  spent  just  for  services  in  2008  (GAO,  2009).  In 
conjunction  with  this  increase  in  defense  procurement  is  the  reduction  of  the  defense 
acquisition  workforce.  The  size  of  the  federal  workforce  decreased  from  2.25  million  in  1990 
to  1.78  million  in  2000  (GAO,  2002).  The  combination  of  the  increasing  defense 
procurement  workload  and  the  decreasing  size  of  the  government  workforce,  along  with  the 
complexities  of  an  arcane  and  convoluted  government  contracting  process,  have  created 
the  perfect  storm — an  environment  in  which  complying  with  government  contracting  policies 
and  adopting  contract  management  best  practices  has  not  always  been  feasible  (Rendon, 
2010).  Between  2001  and  2009,  the  GAO  issued  16  reports  related  to  trends,  challenges, 
and  deficiencies  in  defense  contracting.  During  this  same  time  frame,  the  DoD  Inspector 
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General  (DoDIG)  issued  142  reports  on  deficiencies  in  the  DoD  acquisition  and  contract 
administration  processes.  These  reports  have  identified  poor  contract  planning,  contract 
administration,  and  contractor  oversight  as  just  some  of  the  critically  deficient  areas  in  DoD 
contract  management.  Because  of  these  deficiencies,  the  GAO  has  identified  contract 
management  as  a  “high  risk”  area  for  the  federal  government  since  1990  and  continues  to 
identify  it  as  high  risk  (GAO,  2013). 

As  the  DoD’s  services  acquisition  continues  to  increase  in  scope  and  dollars,  the 
agency  must  give  greater  attention  to  proper  acquisition  planning,  adequate  requirements 
definition,  sufficient  price  evaluation,  and  proper  contractor  oversight  (GAO,  2002).  In  fact, 
as  stressed  in  a  recent  memorandum  for  acquisition  professionals  by  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L],  2010),  improving  the 
efficiency  of  the  acquisition  of  products  and  services  is  of  utmost  importance  to  the  DoD.  In 
some  ways,  the  issues  affecting  services  acquisition  are  similar  to  those  affecting  the 
acquisition  of  physical  supplies  and  weapon  systems.  However,  the  unique  characteristics  of 
services  and  the  increasing  importance  of  services  acquisition  offer  a  significant  opportunity 
for  conducting  research  in  the  management  of  services  acquisition  in  the  DoD. 

Research  Questions 

This  research  project  undertakes  a  focused,  in-depth  study  of  the  services 
acquisition  so  as  to  understand  how  success  of  service  acquisition  contracts  is  being 
defined  and  measured  in  the  Navy.  The  contract  management  process  is  performed  with 
inputs  from  the  different  functional  areas,  such  as  program  management,  contracting, 
financial,  logistics,  and  quality  assurance.  Each  of  these  project  team  members  represents 
different  stakeholders  and  are  therefore  likely  to  have  different  goals  and  objectives.  Hence, 
the  first  research  question  we  investigated  was  as  follows:  How  do  different  stakeholders 
define  successful  services  contracts  within  the  Navy?  To  develop  a  clear  understanding  of 
current  services  acquisition  practices,  we  also  investigated  the  second  research  question: 
How  do  different  stakeholders  measure  services  contracts  within  the  Navy?  Investigating  the 
previous  two  questions  helped  us  develop  recommendations  regarding  the  third  and  final 
research  question:  How  should  the  service  contract’s  success  be  measured?  The  next 
section  provides  a  literature  review  of  some  of  the  management  theories  informing  service 
supply  chain  management,  as  well  as  some  of  our  previous  research  on  DoD  services 
acquisition. 

Literature  Foundation 

The  academic  research  in  the  management  of  services  acquisition  is  founded  on 
several  economic  and  management  theories,  including  agency  theory  (Eisenhardt,  1989), 
transaction  cost  economics  (Williamson,  1979),  contractual  theory  (Luo,  2002),  service 
operations  and  supply  management  (Fitzsimmons  &  Fitzsimmons,  2006),  and  stakeholder 
theory  (Freeman,  1984;  Cleland,1986;  El-Gohary,  Osman,  &  El-Diraby,  2006).  We  refer  the 
reader  to  our  earlier  technical  report  (Apte  &  Rendon,  2013)  for  a  survey  of  prior  academic 
research,  and  we  also  provide  a  summary  of  research  projects  carried  out  by  the  authors  in 
the  area  of  services  supply  chain. 

We  have  addressed  the  need  for  research  in  this  increasingly  important  area  of 
services  acquisition  by  undertaking  six  sponsored  research  projects  over  the  past  six  years. 
The  first  two  research  projects  (Apte,  Ferrer,  Lewis,  &  Rendon,  2006;  Apte  &  Rendon,  2007) 
were  exploratory  in  nature,  aimed  at  understanding  the  types  of  services  being  acquired,  the 
associated  rates  of  growth  in  services  acquisition,  and  the  major  challenges  and 
opportunities  present  in  the  service  supply  chain. 
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The  next  two  research  projects  were  survey-based  empirical  studies  aimed  at 
developing  a  high-level  understanding  of  how  services  acquisition  is  currently  being 
managed  at  a  wide  range  of  Army,  Navy,  and  Air  Force  installations  (Apte,  Apte,  &  Rendon, 
2008;  Apte,  Apte,  &  Rendon,  2009).  The  analysis  of  survey  data  indicated  that  the  current 
state  of  services  acquisition  management  suffers  from  several  deficiencies,  including  deficit 
billet  and  manning  levels  (which  are  further  aggravated  by  insufficient  training  and  the 
inexperience  of  acquisition  personnel)  and  the  lack  of  strong  project-team  and  life-cycle 
approaches.  Our  research  (Apte,  Apte,  &  Rendon,  2010)  also  analyzed  and  compared  the 
results  of  the  primary  data  collected  in  two  previous  empirical  studies  involving  Army,  Navy, 
and  Air  Force  contracting  organizations  so  as  to  develop  a  more  thorough  and 
comprehensive  understanding  of  how  services  acquisition  is  being  managed  within 
individual  military  Services. 

As  a  result  of  these  research  projects  dealing  with  the  service  supply  chain  in  the 
DoD,  we  have  developed  a  comprehensive,  high-level  understanding  of  services  acquisition 
in  the  DoD,  have  identified  several  specific  deficiencies,  and  have  proposed  a  number  of 
concrete  recommendations  for  performance  improvement. 

Based  on  the  foundation  of  the  previously  mentioned  management  theories, 
conclusions  of  the  GAO  and  DoDIG  reports  (Seifert  &  Ermoshkin,  2010),  and  findings  of  our 
own  sponsored  research  projects  on  the  topic,  we  believe  that  the  success  of  service 
acquisition  contracts  is  significantly  influenced  by  four  broadly  defined  factors:  (1)  the  type 
and  quantity  of  services  being  outsourced  and  the  associated  amount  of  acquisition-related 
workload;  (2)  the  characteristics  of  contracts  being  awarded;  (3)  the  capacity  available  to 
carry  out  the  contracting,  project  management,  and  surveillance  work;  and  (4)  various 
management  practices,  such  as  use  of  project  team  or  life-cycle  approaches  and  so  forth.  A 
conceptual  model  indicating  the  interrelationship  among  these  factors  is  shown  in  Figure  1 . 


Figure  1.  Drivers  of  Acquisition  Practices  and  Success  of  Service 

Contracts 
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As  shown  in  the  conceptual  diagram  of  Figure  1 ,  the  contract  characteristics  are 
affected  by  the  type  of  service  being  acquired,  while  the  management  practices  being  used 
are  influenced  by  the  services  being  acquired,  the  contract  characteristics,  and,  more 
importantly,  the  capacity  available  to  perform  the  acquisition  work.  The  success  of  services 
contracts,  in  turn,  is  affected  by  the  previously  mentioned  four  drivers.  Underlying  Figure  1  is 
the  fundamental  question  motivating  our  in-depth  research:  What  drives  the  success  of 
services  contracts?  This  fundamental  question  is,  of  course,  critically  important,  and  yet  it  is 
also  not  one  that  can  be  answered  easily  or  quickly.  We  believe  that,  generally,  in  the  case 
of  questions  related  to  complex  systems,  it  is  preferable  to  break  down  the  overall  system  in 
smaller  parts,  gain  an  understanding  of  the  functioning  of  each  part,  and  then  put  all  the 
pieces  together  to  better  understand  the  overall  system  and  answer  the  fundamental 
question.  That  is  what  we  plan  to  do  in  this  research  by  addressing  three  research 
questions:  (1)  understand  how  the  success  of  services  contracts  is  being  defined  by 
different  stakeholders,  (2)  identify  how  the  success  of  services  contracts  is  currently  being 
measured,  and  (3)  develop  specific  recommendations  on  how  the  success  of  services 
contracts  should  be  measured.  We  address  our  research  methodology  in  the  next  section. 

Research  Methodology 

With  the  assistance  of  our  MBA  thesis  students  (Hagan,  Spede,  &  Sutton,  2012),  we 
developed  and  deployed  a  data  collection  survey  instrument  to  collect  empirical  data  for 
answering  our  research  questions.  The  survey  was  deployed  to  the  various  stakeholders  at 
the  participating  commands.  We  then  analyzed  the  data  using  descriptive  statistics  to 
provide  recommendations  and  conclusions. 

We  developed  and  deployed  a  web-based  survey  using  the  SurveyMonkey  website. 
The  survey  instrument  included  both  demographic  questions  and  core  questions  related  to 
defining  and  measuring  successful  services  contracts.  The  core  questions  were  designed  to 
establish  the  importance  of  different  factors  when  defining  and  measuring  the  success  of 
services  contracts.  These  core  questions  were  related  to  the  contracting  process,  as  well  as 
to  different  outcomes  such  as  cost,  schedule,  and  performance  (Hagan  et  al.,  2012). 

In  terms  of  defining  successful  contracts,  the  core  questions  asked  participants  to 
rank  various  definitions  relating  to  the  four  metrics  (process,  cost,  schedule,  and 
performance)  in  order  of  most  important  (1 )  to  least  important  (5).  We  also  asked 
participants  to  rate  definition  statements  relating  to  process,  cost,  schedule,  and 
performance.  These  questions  use  a  Likert  scale  asking  level  of  agreement,  importance, 
and  amount  of  time  devoted  by  the  participants.  The  Likert  scale  had  a  range  of  1  to  5,  with 
1  representing  a  negative  response  and  5  representing  a  positive  response  (Hagan  et  al., 
2012). 

In  terms  of  measuring  successful  contracts,  the  core  questions  asked  participants  to 
rank  various  measurements  relating  to  the  four  metrics  in  order  of  most  important  (1 )  to  least 
important  (5).  The  last  question  in  the  section  asks  participants  to  rate  on  a  Likert  scale  how 
often  the  organization  conducts  certain  actions  that  pertain  to  the  measurement  of  success 
concerning  process,  schedule,  cost,  and  performance.  Figure  2  reflects  our  survey  question 
approach  (Hagan  et  al.,  2012). 
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Figure  2.  Diagram  of  Survey  Questions 

The  survey  was  deployed  to  the  major  stakeholders  (PMs,  COs,  and  CORs)  at  the 
following  major  contracting  commands:  Fleet  Logistics  Center  (FLC)  Philadelphia,  FLC 
Jacksonville,  FLC  Norfolk,  FLC  Puget  Sound,  FLC  San  Diego,  Naval  Sea  Systems 
Command  (NAVSEA),  Military  Sealift  Command  (MSC),  and  Space  and  Naval  Warfare 
Systems  Command  (SPAWAR;  Hagan  et  al.,  2012). 

Survey  Results  and  Analysis 

In  this  section,  we  present  the  results  of  the  survey  and  discuss  its  major  findings.  As 
mentioned  previously,  the  primary  objective  of  this  research  is  to  empirically  examine  how 
the  success  of  a  service  contract  is  being  defined  and  measured  by  different  stakeholders. 
We  designed  a  survey  containing  19  questions  and  distributed  them  to  the  major 
stakeholders  in  the  services  acquisition  process  to  receive  their  responses.  The  survey  was 
deployed  at  the  eight  Navy  installations  identified  previously.  We  distributed  the  survey  to  a 
total  of  843  respondents  responsible  for  various  acquisition-related  functions.  Specifically, 
we  surveyed  the  following  stakeholders:  program  manager/project  officer,  contract 
officer/contract  specialist,  contracting  officer  representative,  requirements  manager,  financial 
manager,  contractor,  and  customer.  The  survey  questions  included  both  Likert-type  as  well 
as  ranking-type  questions.  The  Likert-type  questions  were  used  to  assess  favorable  or 
unfavorable  responses,  while  the  ranking-type  questions  were  used  to  assess  the  most 
important  responses.  When  we  examine  the  ranking  questions  in  this  section,  the  term 
“most  important”  refers  to  the  number  of  factors  that  received  the  highest  rankings  of  1  or  2. 
We  believe  that  this  is  the  best  way  to  capture  and  succinctly  represent  the  participants’ 
responses.  For  example,  a  COR  may  feel  that  the  outcome-related  factors  are  extremely 
important  and,  therefore,  should  be  given  the  highest  ranking  of  1  every  time.  However,  the 
COR  may  also  believe  that  the  process-related  factors  are  very  important,  too,  and  hence 
may  assign  the  next  highest  rank  of  2  to  those  factors.  Hence,  we  believe  that  the  percent  of 
respondents  giving  a  rank  of  1  or  2  to  a  factor  is  the  most  effective  way  to  capture  and 
represent  the  importance  of  that  factor  while  analyzing  the  data  on  ranking  of  factors. 

The  survey  response  rates  we  experienced  for  different  categories  of  stakeholders 
are  shown  in  Table  1 .  Unfortunately,  we  received  only  a  small  number  of  responses  from 
requirements  managers,  financial  managers,  contractors,  and  customers.  Hence,  their 
responses  are  not  incorporated  in  this  report  for  analysis  purposes.  These  respondents  are 
combined  under  the  “other”  category  in  Table  1 . 
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Table  1.  Survey  Response  Rate 


STAKEHOLDER 

# SURVEYS 
DEPLOYED 

#  SURVEYS 
ANSWERED 

RESPONSE 

RATE 

PROGRAM  MANAGER/PROJECT  OFFICER 

94 

15 

16% 

CONTRACTING  OFFICER 

REPRESENTATIVE 

104 

27 

26% 

CONTRACTING  OFFICER/  CONTRACT 
SPECIALIST 

280 

126 

45% 

AGGREGATE  DATA  (PM,  COR,  PCO) 

478 

168 

35% 

OTHER 

365 

10 

2.7% 

TOTAL 

843 

178 

21% 

We  present  the  survey  results  and  analysis  in  three  sub-sections:  the  first  sub¬ 
section  presents  the  aggregate  data,  the  second  sub-section  presents  the  stakeholder-level 
data,  and  the  third  sub-section  presents  the  service-type  data. 


Survey  Results:  Aggregate  Survey  Data 

Defining  the  Success  of  a  Service  Contract 

In  taking  a  high-level  view  of  our  survey  findings,  we  did  not  differentiate  between 
functional  roles,  DAWIA  levels  of  certification,  type  of  service  being  acquired,  contract  type, 
or  the  organization.  However,  we  did  separate  our  findings  under  the  broad  categories  of 
process  and  outcome.  Outcome  results  included  the  questions  associated  with  cost, 
schedule,  and  performance.  As  shown  in  Table  1,  collectively,  there  were  168  responses 
from  PMs,  CORs,  or  PCOs.  The  Likert  scale  responses  were  assigned  a  value  of  1  through 
5,  with  the  higher  value  representing  a  more  favorable  response  to  a  statement.  A  summary 
of  aggregate  data  about  defining  and  measuring  the  success  of  a  service  contract  is 
presented  in  Tables  2  and  3  of  Appendix  A.  We  examined  the  mean  of  responses  to  each 
set  of  Likert  scale-type  questions.  We  found  that  when  defining  the  success  of  a  services 
contract,  outcomes  are  considered  slightly  more  important  than  processes.  The  overall 
mean  of  responses  related  to  outcomes  was  4.08,  while  process  responses  resulted  in  a 
mean  of  3.97.  Our  findings  are  displayed  graphically  in  Figure  3. 

We  then  separated  our  findings  further  within  the  broad  category  of  outcomes  into 
the  narrower  categories  of  cost,  schedule,  and  performance.  Performance-related  questions 
resulted  in  the  highest  mean  of  4.29,  while  cost-related  questions  produced  a  mean  of  4.03, 
and  schedule-related  questions  produced  a  mean  of  3.93. 

One  hundred  and  sixty-eight  respondents  were  asked  to  rank  different  factors  related 
to  defining  the  success  of  a  service  contract.  These  questions  also  dealt  with  different 
aspects  of  processes  and  outcomes.  Of  the  1 68  respondents,  40%  felt  that  process-related 
factors  were  the  most  important.  Sixty  percent  felt  that  outcome-related  factors  were  the 
most  important.  The  distribution  of  highest  ranked  responses  is  displayed  in  Figure  4. 

Breaking  down  the  outcome-related  factors  further,  15%  of  respondents  felt  that 
cost-related  factors  were  the  most  important,  19%  felt  that  schedule-related  factors  were 
most  important,  and  26%  felt  that  performance-related  factors  were  most  important. 
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Means  of  Aggregate  Stakeholder 
Definitions  of  Success  (n=168) 
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Figure  3.  Means  of  Aggregate  Stakeholder  Definitions  of  Success 
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Figure  4.  Aggregate  Stakeholder  Ranking  of  Definitions  of  Success 
Measuring  the  Success  of  a  Service  Contract 

Our  survey  also  requested  that  participants  rate  on  the  Likert  scale  the  various 
degrees  of  importance,  and  the  extent  to  which  they  agreed  or  disagreed  with,  various 
factors  when  considering  how  they  measure  the  success  of  a  service  contract.  Again,  these 
factors  related  to  either  processes  or  outcomes.  The  overall  Likert  scale  mean  with  relation 
to  processes  was  2.48,  and  the  outcomes  displayed  an  overall  mean  of  3.71 .  Clearly 
outcomes  are  deemed  more  important  by  our  participants  as  a  whole.  Our  findings  are 
displayed  graphically  in  Figure  5. 
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If  we  look  at  the  distinct  factors  within  outcome  of  cost,  schedule,  and  performance, 
the  overall  Likert  means  were  3.96,  3.84,  and  3.30,  respectively. 

One  hundred  and  sixty-eight  respondents  were  asked  to  rank  different  factors  related 
to  measuring  the  success  of  a  service  contract.  Of  the  168  respondents,  46%  felt  that 
process-related  factors  were  the  most  important.  Fifty-four  percent  felt  that  outcome-related 
factors  were  the  most  important.  The  distribution  of  highest  ranked  responses  is  displayed  in 
Figure  6. 

Breaking  down  the  outcome-related  factors  further,  19%  of  respondents  felt  that 
cost-related  factors  were  the  most  important,  12%  felt  that  schedule-related  factors  were 
most  important,  and  23%  felt  that  performance-related  factors  were  most  important. 

Means  of  Aggregate  Stakeholder 
Measurements  of  Success 
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Figure  5.  Means  of  Aggregate  Stakeholder  Measurements  of  Success 
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Figure  6.  Aggregate  Stakeholder  Ranking  of  Measurements  of  Success 
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Analysis  of  Aggregate  Survey  Data 

The  findings  from  the  analysis  of  aggregate  survey  data  show  that  when  asked  to 
respond  on  a  Likert  scale,  different  stakeholders  find  all  aspects  of  processes  and  outcomes 
important  when  defining  the  success  of  a  service  contract.  The  means  of  the  responses  we 
collected  are  very  close,  and  it  does  not  seem  that,  as  a  whole,  our  population  favors 
process  or  outcome  when  defining  success.  Perhaps  this  is  due  to  the  nature  of  Likert  scale 
questions.  When  asked  if  something  such  as  cost  overruns,  major  milestones,  or  a  lack  of 
protests  is  important,  all  stakeholders  will  invariably  say  yes.  That  is  why  the  overall  mean  of 
all  responses,  for  both  outcomes  and  processes,  is  fairly  high  at  4.03.  When  forced  to  rank, 
the  responses  differ  and  outcome-related  responses  received  a  high  rank  of  1  or  2  60%  of 
the  time.  This  is  because  outcomes  such  as  keeping  on  schedule  and  budget  adherence  are 
easy  to  understand  and  define.  Process-related  factors  such  as  administration  and 
communication  are  relatively  harder  to  quantify. 

The  findings  also  demonstrate  that  when  measuring  the  success  of  a  service 
contract,  all  stakeholders  tend  to  focus  on  outcomes  and  do  not  take  into  consideration  the 
processes;  this  was  true  for  both  Likert-scale  responses  and  ranking  responses.  This  is  very 
evident  in  the  Likert-scale  responses,  where  none  of  the  process-related  factors  showed  a 
mean  of  3  or  more.  When  forced  to  rank  the  different  factors  with  respect  to  measuring 
success,  the  results  were  similar  to  defining  success,  with  56%  of  “most  important” 
responses  falling  under  the  outcomes  category. 

In  general  our  findings  from  the  “other”  category  mirrored  our  aggregate  results. 
Although  there  were  only  10  responses,  all  felt  that  outcomes  were  the  most  important  factor 
when  defining  and  measuring  the  success  of  a  service  contract.  We  found  that  our 
stakeholders  in  this  category  rated  and  ranked  processes  extremely  low  in  both  defining  and 
measuring  the  success  of  a  service  contract.  This  is  because  these  stakeholders  are  not 
terribly  burdened  by  administration  and  other  process-related  factors,  so  they  feel  that  these 
factors  are  not  important.  For  example,  a  contractor  or  end  user  does  not  necessarily 
conduct  market  research  or  choose  the  appropriate  contract  type.  However,  they  are  very 
concerned  with  staying  within  cost,  keeping  up  with  schedule,  and  maintaining  a  high  level 
of  performance. 

Survey  Results:  Stakeholder-Level  Data 

As  a  starting  point  in  examining  how  different  stakeholders  define  and  measure  the 
success  of  a  service  contract,  we  performed  a  statistical  analysis  of  the  data  to  determine 
whether  there  were  significant  differences  between  the  ratings  on  the  Likert  scale  across  the 
major  stakeholders.  We  first  performed  an  F-test  for  sample  variances  to  determine  the 
appropriate  f-test  to  perform.  In  all  instances,  we  found  that  there  was  an  equal  variance 
among  stakeholders.  The  only  statistically  significant  difference  was  between  the  CORs  and 
COs/specialists  when  measuring  success.  This  could  be  due  to  the  fact  that  CORs  view 
communication  and  other  processes  as  key  factors  when  measuring  the  success  of  a 
service  contract.  The  COR  is  also  likely  to  view  a  protest  as  a  serious  issue  when  measuring 
success  because  it  results  in  a  delay  of  execution  and  CORs  cannot  perform  their  duties. 
Otherwise,  there  was  no  statistically  significant  difference  between  any  other  of  the 
stakeholders  on  the  Likert  scale.  We  discuss  in  the  next  section  the  results  of  the  analysis  of 
stakeholder-level  data. 

Analysis  of  Stakeholder-Level  Data 

Consistent  with  the  abovementioned  results  of  statistical  analysis,  we  found  that 
PMs,  CORs,  COs,  and  contract  specialists  all  agree  that  outcome  is  slightly  more  important 
than  processes  based  on  participants’  ratings  of  separate  factors  on  a  Likert  scale.  Each 
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functional  role  rated  outcome  slightly  over  4.00,  while  rating  processes  just  below  4.00.  The 
mean  of  the  functional  roles  combined  was  3.94  for  processes,  and  4.1 1  for  outcomes. 
Within  outcome,  performance-related  factors  received  the  highest  average  rating,  while 
schedule-related  factors  received  the  lowest  average  rating.  All  functional  roles  showed  an 
upward  trend  from  schedule,  to  cost,  to  performance.  A  comparison  of  our  Likert  scale 
findings  for  defining  success  across  functional  roles  is  displayed  graphically  in  Figure  7. 


When  stakeholders  were  asked  to  rank  different  factors  concerning  their  definition  of 
success,  we  found  that  there  was  clear  agreement  that  outcomes  are  more  important  than 
processes.  There  was,  however,  some  disagreement  within  the  outcome  factors  of  cost, 
schedule,  and  performance.  CORs  felt  that  cost  was  the  most  important  factor,  while  PMs, 
COs,  and  specialists  placed  performance  at  the  top  of  their  rankings.  Examined  collectively, 
the  major  stakeholders  provided  168  responses  when  ranking  their  definition  of  the  success 
of  a  service  contract.  Sixty  percent  of  respondents  felt  that  outcome-related  factors  were 
most  important,  while  40%  felt  that  process-related  factors  were  the  most  important  when 
defining  success.  The  distribution  of  highest  ranked  responses  is  displayed  in  Figure  8. 
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Figure  7.  Definitions  of  Success  Across  Major  Stakeholders 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-373- 
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Figure  8.  Major  Stakeholder  Ranking  of  Definitions  of  Success 


According  to  the  survey  data,  stakeholders  also  tend  to  measure  success  in  almost 
the  same  way.  When  asked  to  rate  different  factors  on  the  Likert  scale  related  to 
stakeholders’  measures  of  success,  all  respondents  agreed  that  outcomes  far  outweigh 
processes.  When  looking  at  the  mean  across  stakeholders,  processes  received  a  rating  of 
2.56,  while  outcomes  received  a  rating  of  3.78.  Within  outcome-related  factors,  stakeholders 
showed  an  upward  trend  from  performance,  to  schedule,  to  cost.  A  comparison  of  our 
findings  for  defining  success  on  the  Likert  scale  across  functional  roles  is  displayed 
graphically  in  Figure  9. 

Our  ranking  data  shows  that,  again,  major  stakeholders  prefer  outcome-related 
factors  when  measuring  the  success  of  service  contracts.  When  examined  in  aggregate,  the 
major  stakeholders  provided  168  responses  to  our  ranking  questions.  Of  these  responses, 
43%  of  respondents  felt  process  factors  were  most  important,  while  57%  were  in  favor  of 
factors  related  to  outcomes.  The  distribution  of  highest  ranked  responses  is  displayed  in 
Figure  10. 
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Figure  9.  Measurement  of  Success  Across  Major  Stakeholders 
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Figure  10.  Major  Stakeholder  Ranking  of  Measurements  of  Success 


The  Likert  scale  responses  for  definitions  of  success  were,  again,  relatively  high,  and 
this  was  due  to  the  reason  explained  earlier.  It  is  interesting  that  in  both  defining  and 
measuring  success,  CORs  ranked  cost  highest  out  of  the  three  stakeholders. 


Another  interesting  result  is  that  COs  tended  to  place  nearly  equal  importance  on 
process  and  outcomes  when  forced  to  rank  factors  concerning  measuring  success.  This  is 
probably  due  to  the  administrative  nature  of  the  COs’  role.  For  example,  their  functional  role 
has  to  deal  with  modifications,  COR  reports,  and  exercising  options.  The  other  functional 
roles  of  PMs  and  CORs  are  not  overly  concerned  with  processes  and  are  focused  on  the 
requirement  and  outcomes.  The  data  reflect  this  fact. 
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It  is  interesting  to  note  that  every  demographic  consistently  rated  processes 
significantly  higher  on  the  Likert  scale  when  defining  success  versus  measuring  success. 

We  feel  that  this  is  because  stakeholders  view  measures  as  a  tangible  entity  associated  with 
post-award  functions.  Measures  such  as  cost,  schedule,  and  performance  are  fairly 
straightforward  inasmuch  as  a  goal  is  either  met  or  not.  Processes  such  as  communication 
flow  and  overall  management  are  more  obscure  and  subjective.  The  stakeholders  rated 
processes  higher  for  defining  success  because  they  are  closely  associated  with  mainly  pre¬ 
award  functions.  Processes  such  as  choosing  the  correct  contract  type  and  appropriately 
evaluating  the  proposal  are  crucial  for  success.  Because  these  are  pre-award  activities,  it  is 
easier  to  define  success  rather  than  measure  it. 

Summary,  Conclusions,  and  Recommendations 

Summary 

The  DoD’s  obligations  on  contracts  have  more  than  doubled  between  fiscal  years 
2001  and  2008  to  over  $387  billion,  with  over  $200  billion  spent  just  for  services  in  2008 
(GAO,  2009).  In  conjunction  with  this  increase  in  defense  procurement  is  the  reduction  of 
the  defense  acquisition  workforce.  The  combination  of  the  increasing  defense  procurement 
workload  and  the  decreasing  size  of  the  government  workforce,  along  with  the  complexities 
of  an  arcane  and  convoluted  government  contracting  process,  have  created  the  perfect 
storm — an  environment  in  which  complying  with  government  contracting  policies  and 
adopting  contract  management  best  practices  has  not  always  been  feasible  (Rendon,  2010). 
The  contract  management  process  is  performed  with  inputs  from  the  different  functional 
areas,  using  a  cross-functional  team  or  integrated  project  team  (IPT)  structure.  Each  of 
these  project  team  members  represents  the  stakeholders,  and  their  different  goals  and 
objectives.  The  first  research  question  we  investigated  was  as  follows:  How  do  different 
stakeholders  define  successful  services  contracts  within  the  Navy?  To  develop  a  clear 
understanding  of  current  services  acquisition  practices,  we  also  investigated  a  second 
research  question:  How  do  different  stakeholders  measure  services  contracts  within  the 
Navy?  Investigating  the  above  two  questions  helped  us  develop  recommendations 
regarding  the  third  and  final  research  question:  How  should  the  service  contract’s  success 
be  measured? 

Conclusions 

On  the  aggregate  level,  our  research  indicated  that,  when  defining  a  successful 
service  contract,  stakeholders  considered  outcomes  (in  the  order  of  performance,  cost,  and 
schedule)  slightly  more  important  than  processes.  Stakeholders  also  ranked  outcome- 
related  factors  as  most  important.  On  the  aggregate,  our  research  indicated  that,  when 
measuring  a  successful  service  contract,  stakeholders  considered  outcomes  (in  the  order  of 
cost,  schedule,  and  performance)  more  important  than  processes.  Stakeholders  also  ranked 
outcome-related  factors  as  most  important. 

On  the  stakeholder  level,  our  research  indicated  that,  when  defining  a  successful 
service  contract,  PMs,  CORs,  and  COs  considered  outcomes  (in  the  order  of  performance, 
cost,  and  schedule)  slightly  more  important  than  processes.  PMs,  CORs,  and  COs  also 
ranked  outcome-related  factors  as  most  important.  On  the  stakeholder  level,  our  research 
indicated  that,  when  measuring  a  successful  service  contract,  PMs,  CORs,  and  COs 
considered  outcomes  (in  the  order  of  performance,  schedule,  and  cost)  more  important  than 
processes.  PMs,  CORs,  and  COs  also  ranked  outcome-related  factors  as  most  important. 

Recommendations 

Our  research  findings  have  several  implications  for  the  Navy,  as  well  as  the  DoD.  All 
stakeholders  surveyed  identified  and  ranked  outcome-related  factors  as  more  important 
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than  process-related  factors,  in  both  defining  and  measuring  the  success  of  service 
contracts.  This  may  be  because  outcome-related  factors  (cost,  schedule,  and  performance) 
are  more  easily  defined  and  measured  using  available  metrics,  compared  to  contracting 
processes,  which  are  more  difficult  to  define,  and  many  agencies  have  no  available  metrics. 
However,  as  discussed  in  the  earlier  sections  of  this  paper,  many  of  the  contracting 
deficiencies  identified  by  the  GAO  and  DoDIG  are  related  to  contracting  processes,  such  as 
conducting  market  research,  determining  item  commerciality,  selecting  contract  type, 
negotiating  fair  and  reasonable  prices,  and  monitoring  contractors  through  surveillance. 
Thus,  our  first  recommendation  is  that  the  U.S.  Navy  develop  and  implement  process- 
related  metrics  to  define  and  measure  critical  contracting  processes,  such  as  conducting 
market  research,  determining  item  commerciality,  selecting  contract  type,  negotiating  fair 
and  reasonable  prices,  and  monitoring  contractors. 

Our  literature  review  identified  that  acquisition  stakeholders  (PMs,  CORs,  and  COs) 
have  different  procurement  goals  and  objectives,  and  these  goals  and  objectives  may  in  fact 
conflict  with  each  other.  Our  second  recommendation  is  that  the  U.S.  Navy  should  establish 
internal  controls  to  ensure  the  contracting  processes  are  being  followed  and  that  the 
different  stakeholders  place  sufficient  importance  on  the  value  of  these  contracting 
processes. 

Finally,  as  previous  research  has  determined  that  contracts  are  only  as  successful  as 
the  processes  used  to  plan,  award,  and  administer  these  contracts,  our  final 
recommendation  is  for  the  U.S.  Navy  to  implement  a  program  for  continuously  assessing  its 
contracting  process  capability  and  using  the  assessment  results  to  improve  its 
organizational  contract  management  process  capability.  Once  the  U.S.  Navy,  as  well  as  the 
DoD,  implement  contracting  process-related  metrics  to  define  and  measure  services 
contracts,  internal  controls  to  ensure  contracting  process  compliance,  and  periodical 
assessments  of  organizational  contracting  process  capability,  the  importance  of  process- 
related  factors  in  defining  and  measuring  the  success  of  service  contracts  will  increase 
among  stakeholders  and  thus  start  addressing  some  of  the  contracting  deficiencies 
identified  by  the  GAO  and  the  DoDIG. 
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